
UTILITY 
PATENT APPLICATION 
TRANSMITTAL 



(Only for new nonprovisional applications under 37 CFR 1.53(b)) 



First Name 


i r*> S—jg. 

i Inventor or Application Identifier ^^rT 


TOSHIKAZU OHSHIMA 


i Q^sn^ 



i° APPLICATION ELEMENTS 

See MPEP chapter 600 concerning utility patent application contents. 



ADDRESS TO: 



Assistant Commissioner for Patents 
Box Patent Application 
Washington, DC 20231 



Fee Transmittal Form 



(Submit an original, and a duplicate for fee processing) 

2. (~X~I Specification Total Pages I 45 I 

3. |~~X~l Drawing(s) (35 USC 11 3) Total Sheets 

4. m 



Microfiche Computer Program (Appendix) 



Oath or Declaration 
X 



1£ 



Total Pages 



7. Nucleotide and/or Amino Acid Sequence Submission 
(if applicable, all necessary) 

a. | | Computer Readable Copy 

b. 1 1 Paper Copy (identical to computer copy) 

c. 1 | Statement verifying identity of above copies 



Newly executed (original or copy) 

b. | | Unexecuted for information purposes 

c. | | Copy from a prior application (37 CFR 1 .63(d)) 

(for continuation/divisional with Box 17 completed) 
[Note Box 5 below] 

i. [ | DELETION OF INVENTOR(S) 

Signed Statement attached deleting inventor(s) 
named in the prior application, see 37 CFR 
1.63(d)(2) and 1.33(b). 

Incorporation By Reference (useable if Box 4c is checked) 
The entire disclosure of the prior application, from which a copy of the 
oath or declaration is supplied under Box 4c, is considered as being 
part of the disclosure of the accompanying application and is hereby 
incorporated by reference therein. 



ACCOMPANYING APPLICATION PARTS 



8.H 

*□ 

10. □ 

iiO 

12. □ 

13. [~xl 

14. □ 

15. □ 

16. □ 



Assignment Papers (cover sheet & documenf(s)) 

Power of Attorney 



37 CFR 3.73(b) Statement I j 
(when there is an assignee) 



English Translation Document (if applicable) 



Information Disclosure 
Statement (IDS)/PTO-1449 

Preliminary Amendment 



I [ Copies of IDS 
' ' Citations 



Return Receipt Postcard (MPEP 503) 
(Should be specifically itemized) 

Small Entity I j Statement filed in prior application 
Statements) Status still proper and desired 

Certified Copy of Priority Document(s) 
(if foreign priority is claimed) 



17. If a CONTINUING APPLICATION, check appropriate box and supply the requisite information: 

I | Continuation j I Divisional I | Continuation-in-part (CIP) of prior application No. . 



18. CORRESPONDENCE ADDRESS 



City 



Customer Number or Bar Code Label 



05514 

! {insert Customer No. or Attach bar code label here) 



State 



□ 



or j | Correspondence address below 



Zip Code 



DC_MAIN 16252 v1 



Country 




Telephone 




Fax 





CLAIMS 


(1)FOR 


(2) NUMBER FILED 


(3) NUMBER EXTRA 


(4) RATE 


(5) CALCULATIONS 




TOTAL CLAIMS 
(37 CFR 1.16(c)) 


58-20 = 


38 


X$ 18.00 = 


$ 684.00 




INDEPENDENT 
CLAIMS (37 cfr 1.16(b)) 


7-3 = 


4 


X$ 78.00 = 


$ 312.00 




MULTIPLE DEPENDENT CLAIMS (if applicable) (37 CFR 1.16(d)) 


$ 260.00 = 


$ 0.00 






BASIC FEE 

(37 CFR 1.16(a)) 


$ 690.00 




Total of above Calculations = 


$ 1686.00 




Reduction by 50% for filing by small entity (Note 37 CFR 1.9, 1 .27, 1.28). 






TOTAL = 


$ 1686.00 



19. Small entity status 

a. I | A Small entity statement is enclosed 

I | A small entity statement was filed in the prior nonprovisional application and such status is still proper 
D - I I and desired. 

c. | I Is no longer claimed. 

20. | X | A check in the amount of $ 1 686.00 to cover the filing fee is enclosed. 

21 . j x I A check in the amount of $ 40.00 to cover the recordal fee is enclosed. 

22. The Commissioner is hereby authorized to credit overpayments or charge the following fees to Deposit Account 
No. 06-1205: 

a. [~x] Fe es required under 37 CFR 1 .16. 

b. I I Fees required under 37 CFR 1 .17. 

c. \Z3 Fees required under 37 CFR 1.18. 



SIGNATURE OF APPLICANT, ATTORNEY, OR AGENT REQUIRED 


NAME 


Brian L. Kfock - Reg. No. 36,570 


SIGNATURE 




DATE 


February 22, 2000 



BLK\cmv 



DC_MAIN 16252 v1 



TITLE OF THE INVENTION 



USER INTERFACE APPARATUS , USER INTERFACE METHOD, GAME 
APPARATUS, AND PROGRAM STORAGE MEDIUM 

5 

FIELD OF THE INVENTION 
The present invention relates to a user interface 
apparatus and method, and a game apparatus, which 
generate an instruction (or command) by adequately 
10 detecting player's (user's) actions in, e.g., an action 
game . 



BACKGROUND OF THE INVENTION 
A current personal computer system uses as a user 
15 interface a pointing device such as a mouse, track pad, 
or the like. However, the user must hold the mouse and 
slide it on a given surface. On the other hand, the 
user must rub against the surface of the track pad with 
his or her hand. Thus, these pointing devices limit 
20 user's actions. A GUI (Graphical User Interface) used 
in the personal computer system or the like is that for 
a two-dimensional space, and is not suitable for that 
in a three-dimensional space. 

For these reasons, it is a common practice in the 
25 technical field of VR (Virtual Reality) or AR 

(Augmented Reality) to input commands to the system by 



switch operations of an input device which is held by a 
user's (player's) hand and has switch buttons, and the 
like. 

In the prior art in which commands are input by 
switch operations on the input device having button 
switches and the like, the number of types or number of 
commands (instructions) is limited by the number of 
buttons. If the number of types or number of commands 
(or instructions) is increased, the number of buttons 
increases inevitably. As a result, the input device 
becomes large in size, and the load on the user 
(player) becomes heavier as he or she must learn the 
button positions. 

Learning the button positions imposes a heavy 
load on the user since the command contents do not 
match the button positions. To put it differently, 
expressing various command contents (e.g., "forward 
movement", "backward movement", "stop", and the like) 
by one operation, i.e., depression of a button (or a 
switch) is difficult if not impossible. 

On the other hand, in the VR (Virtual Reality) or 
AR (Augmented Reality) field, a device for simulating 
user's (player's) hand actions has been proposed. For 
example, in one technique, a sensor for detecting the 
bent angle of a joint of a finger is attached to a hand 
of the user, and a CG (computer graphic) image is 



generated in correspondence with the bent angle of the 
finger detected by that sensor. However, this 
technique aims at simulating hand actions of the user, 
and it is impossible to apply this technique to that 
5 for recognizing the user (player) instructions (or 
commands) in practice. 

In this technique, for example, when the user 
stretches the arm forward, a CG image with the arm 
stretched forward is generated, and display of such 

10 image can be interpreted to be a result of a user 

instruction or forward stretching of the arm in the 
broad sense. However, if user commands are generated 
by only hand actions, every hand actions are unwantedly 
interpreted as commands, and such interface has poor 

15 reliability. 

The present invention has been made to solve the 
conventional problems and has as its object to provide 
a user interface apparatus, user interface method, and 
game apparatus, to which the user (player) can easily 

20 sensuously become accustomed, and which can accurately 
recognize instructions (commands) that the user 
(player) intended. 

SUMMARY OF THE INVENTION 
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In order to achieve the above object, a user 
interface apparatus according to the present invention 
comprises : 

a first sensor attached to a first portion of a 
5 body of a user; 

a second sensor attached to a second portion 
different from the first portion; 

means for generating action information of the 
second portion on the basis of a relative position of 
10 the second portion with respect to the first portion, 
which is detected by the first and second sensors; and 

determination means for determining a user 
instruction corresponding to the generated action 
information . 

15 Also, a game apparatus according to the present 

invention comprises: 

a first sensor for detecting a location/posture 
of a head of a player; 

a second sensor for detecting a location/posture 
20 of a hand or arm; 

means for estimating an action of the player on 
the basis of a relative location/posture of the hand or 
arm with respect to the location/posture of the head, 
which are detected by the first and second sensors; and 
25 means for outputting a player command 

corresponding to the estimated action. 
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Likewise, according to the present invention, a 
user interface method for outputting a user instruction 
to a predetermined apparatus or program, comprises: 

the step of detecting a location of a first 
5 portion of a body of a user and a location of a second 
portion different from the first portion using first 
and second sensors attached to the user; and 

the step of determining a user instruction on the 
basis of a relative position of the second portion with 
10 respect to the first portion, which are detected by the 
first and second sensors, and outputting the determined 
user instruction to the apparatus or program. 

In these inventions, when the user (player) 
expresses his or her intention or instruction by his or 
15 her own action, he or she often expresses it by a 

combination of actions of a plurality of his or her 
body portions. Such combination of actions can be 
estimated by detecting the location of the second 
portion relative to the first portion, as described 
20 above. 

The user or player normally expresses an action 
by a relative location with respect to his or her 
f ield-of-view direction. The f ield-of-view direction 
is roughly determined by the head location. Hence, 
25 according to a preferred aspect of the present 
invention, the first portion is a head. 



- 5 - 



According to a preferred aspect of the present 
invention, the second portion is a hand. This is 
because it is easiest for the user to express an action 
by hand. 

5 According to a preferred aspect of the present 

invention, the first (second) sensor detects a location 
or location/posture of the first (second portion. 

User's (player's) action is preferably recognized 
as a state change. Hence, according to a preferred 

10 aspect of the present invention, information which 

pertains to a state change of the second portion with 
respect to a location of the first portion, and 
information which pertains to a location change 
velocity of the state change are detected. 

15 According to a preferred aspect of the present 

invention, information which pertains to a state change 
of the second portion with respect to a location of the 
first portion, and information which pertains to a 
location change acceleration of the state change are 

2 0 detected. 

According to a preferred aspect of the present 
invention, information which pertains to a posture of 
the second portion with respect to a posture of the 
first portion is detected. 

25 According to a preferred aspect of the present 

invention, information which pertains to a moving 
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direction of a location of the second portion with 
respect to a posture of the first portion is detected. 

Various methods of analyzing an action state are 
available. According to claim 10 or 25 as a preferred 
5 aspect of the present invention, a value of the 

relative position of the second portion with respect to 
the first portion, a plurality of state values which 
are defined in advance as a result of transition of the 
value, and a plurality of different user instruction 
10 values corresponding to the plurality of state values 
are stored. 

A user instruction or command is composed of a 
plurality of portions in practice, and it is 
advantageous for a program to individually execute 

15 those portions of the instruction or command. Hence, 
according to a preferred aspect of the presnt9 
invention, the determined user instruction or command 
is decomposed into a plurality of instruction operands, 
and the operands are output. 

20 According to another preferred aspect of the 

present invention, when it is determined that a 
relative relationship between a location/posture of a 
head detected by the first sensor, and a 
location/posture of a hand detected by the second 

25 sensor indicates an action of the user whose line of 
sight is pointing to a predetermined portion of the 
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hand, a user instruction for outputting an operation 
guidance (so-called on-line HELP) is generated. 

According to a preferred aspect of the present 
invention, a third sensor for detecting a bent angle of 
5 a finger is further used. 

When the present invention is applied to the game 
apparatus, a display device is preferably used. Hence, 
the game apparatus according to a preferred aspect of 
the present invention comprises display means for 
10 displaying an image of a game scene in front of the 
head of the player for the player. Furthermore, the 
display means is preferably a head-mounted display. 

When the present invention is applied to the game 
apparatus, three steps including preparation, execution, 
15 and return steps, are preferably prepared for the 

player command. According to a preferred aspect of the 
present invention, the steps are respectively defined 
as : 

a preparation action step for the first player 
20 command when the location of the hand of the player 
moves to a position behind a position in front of a 
face of the player; 

an execution step for the first player command 
when the location of the hand moves forward from a rear 
25 position after the preparation action; and 



a return action step for the first player command 
when the location of the hand returns to a position of 
the face of the player after the forward movement. 

Note that the above object can also be achieved 
5 by a program storage medium that stores a computer 
program for implementing the aforementioned user 
interface method. 

Other features and advantages of the present 
10 invention will be apparent from the following 

description taken in conjunction with the accompanying 
drawings, in which like reference characters designate 
the same or similar parts throughout the figures thereof. 

15 BRIEF DESCRIPTION OF THE DRAWINGS 

The accompanying drawings, which are incorporated 
in and constitute a part of the specification, 
illustrate embodiments of the invention and, together 
with the description, serve to explain the principles 
20 of the invention. 

Fig. 1 is a view for explaining the layout of 
sensors attached to a player who joins in a game system 
according to an embodiment of the present invention; 
Fig. 2 is a block diagram for explaining the 
25 arrangement of the game system according to the 
embodiment of the present invention; 



Fig. 3 is a view for explaining the relationship 
between the head and hand coordinate systems when 
viewed from a reference coordinate system as the 
center; 

5 Fig. 4 is a view for explaining the relationship 

between the head and hand coordinate system when viewed 
from the head coordinate system as the center; 

Fig. 5 is a view for explaining setups of a 
coordinate system of a head sensor 100 according to the 
10 scheme shown in Fig. 4; 

Fig. 6 is a view for explaining setups of a 
coordinate system of a hand sensor 200 according to the 
scheme shown in Fig. 4; 

Fig. 7 is a block diagram for functionally 
15 explaining the arrangement of a command generation unit 
2000; 

Fig. 8 is a flow chart for explaining the overall 
sequence of an action analysis section 2002; 

Fig. 9 is a view for explaining state transition 
20 in the game system according to the embodiment of the 
present invention; 

Fig. 10 is a flow chart for explaining details of 
step S8 in Fig. 8; 

Fig. 11 is a flow chart for explaining details of 
25 step S100 in Fig. 10; 
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Fig. 12 is a flow chart for explaining details of 
step S200 in Fig. 10; 

Fig. 13 is a flow chart for explaining details of 
step S300 in Fig. 10; 
5 Fig. 14 is a flow chart for explaining details of 

step S400 in Fig. 10; 

Fig. 15 is a flow chart for explaining details of 
step S500 in Fig. 10; 

Fig. 16 shows a state example upon outputting a 
10 "loading state"; 

Fig. 17 shows a state example upon outputting a 
"defense state"; and 

Fig. 18 shows a state example upon outputting a 
"firing state". 

15 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
A game apparatus to which a user interface of the 
present invention is applied will be described in 
detail hereinafter with reference to the accompanying 

20 drawings. As will be apparent from the following 

description of the embodiment, the user interface of 
the present invention can be applied to a personal 
computer system or workstation system for providing a 
VR or AR environment to the user in addition to the 

25 game apparatus by modifying the present invention 
within the scope of the invention. 
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Fig. 1 shows a user 1000 who manipulates the game 
apparatus of this embodiment. A magnetic sensor 100 as 
an example of a location/posture detection means is 
attached to the head of the user 1000, and a magnetic 
5 sensor 200 is also attached to a hand. The magnetic 

sensors output electrical signals including information 
of a three-dimensional location (x, y, z) and posture 
(roll, pitch, yaw) of the portions where they are 
attached. 

10 Fig. 2 shows the game apparatus system of this 

embodiment. Referring to Fig. 2, the game system of 
this embodiment has a command generation unit 2000 for 
receiving signals from the two sensors and converting 
them into a command, and a game apparatus 3000. The 

15 command generation unit 2000 analyzes an action of the 
user 1000 on the basis of the output signals from the 
sensors 100 and 200, and generates a command in 
accordance with the analysis result. The generated 
command is sent to the game apparatus 3000, which 

20 executes the command, i.e., the game progresses. 

The game apparatus 3000 displays progress of the 
game on a display unit 4000 such as a display monitor, 
a head-mounted display (HMD) the user wears on his or 
her head, or the like, and the user can make actions 

25 such as loading, shooting, defense, and the like with 
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respect to a "target" as a CG image generated by the 
game apparatus 300 within the displayed virtual space. 

The arrangement and operation of the command 
generation unit 2000 will be described below. As shown 
5 in Fig. 2, the unit 2000 has location/posture 

measurement sections 2001a and 2001b, action analysis 
section 2002, and command conversion section 2003. 

The location/posture measurement sections 2001a 
and 2001b convert electrical signals input from the 

10 location sensors 100 and 200 into six each coordinate 
values representing the locations/postures of the 
individual portions, and send them to the action 
analysis section 2002. The location/posture 
measurement sections 2001a and 2001b have internal 

15 clocks to measure measurement times t in addition to 
the six each coordinate values of the 

locations/postures, and output the measured times to 
the action analysis section 2002. 

The action analysis section 2002 analyzes an 

20 action in consideration of not only the absolute 

locations/postures of the head and hand but also their 
relative location/posture. This is for the following 
reason. That is, a person normally expresses his or 
her own intention by actions of a plurality of body 

25 portions. In other words, when the location/posture of 
only one portion are detected and the user's intention 



- 13 - 



is estimated based on them, many errors are produced 
even when the detected location/posture have high 
precision. For example, as for a forward stretching 
action of the arm, a forward stretching action of the 
5 arm with the head facing forward has a meaning 
different from that with the head facing another 
direction. For example, if the action "to stretch the 
arm forward" means "attack", since attack is normally 
done with the player's head facing the target (i.e., in 

10 the stretching direction of the arm) , the player may 
intend an action other than attack if he or she 
stretched the arm with the head facing another 
direction. Hence, if no posture is taken into 
consideration, a forward stretching action of the arm 

15 with the head facing another direction is highly likely 
to be erroneously recognized as attack. For this 
reason, in this embodiment, the estimation precision of 
the user's intention is improved by taking relative 
location/posture (e.g., the location/posture of the 

20 hand relative to those of the head) into consideration. 
Fig. 3 shows the relationship between the 
locations/postures of the head and hand in a reference 
coordinate system, and Fig. 4 shows the relationship 
between the locations /postures of the head and hand in 

25 a head coordinate system. Figs. 5 and 6 respectively 
show the head and hand coordinate systems. 



- 14 - 



Fig. 7 shows the functions of the action analysis 
section 2002. The functions shown in Fig. 7 are 
implemented by the control sequence according to the 
flow chart shown in Fig. 8. According to the flow 
5 chart shown in Fig. 8, after starting the process, 

three-dimensional locations (L head and L hand ) and postures 
P(Phead and P hand ) of the head and hand are input from the 
location/posture measurement sections 2001 in step S2, 
and a location and posture (L' hand and P' hand ) of the hand 

10 relative to the head are computed based on the input 
locations and postures in step S4. More specifically, 
a coordinate transform matrix M hand from the reference 
coordinate system into the hand coordinate system is 
computed on the basis of the hand location and posture 

15 (Lhand an d P hand) ' a coordinate transform matrix M head from 
the reference coordinate system into the head 
coordinate system is computed on the basis of the 
location and posture (L head and P head ) of the head, and 
based on these matrices, a coordinate transform matrix 

20 M' hand from the head coordinate system into the hand 
coordinate system is computed by: 

M\ and = M hand -M head - X (1) 
where M head _1 is the inverse matrix of M nead . The relative 
location and posture (L' nand , P' hand ) of the hand with 

25 reference to the head (i.e., in the head coordinate 
system) can be easily derived from M' hand . 
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In step S6, change velocity V hand (time derivative 
of L 'hand) of tne hand location L' hand , and acceleration 
A hand (time derivative of velocity V hand ) are computed. 
Note that the currently measured location and posture 
5 (L'hand^ p 'hand) of the hand relative to the head are 

stored in a predetermined memory for the next velocity 
and acceleration computations in step S6. In this case, 
data for two previous measurements suffice. In this 
specification, a combination of information of the 

10 location, posture, time, and the like for the head, and 
those for the hand input from the location/posture 
measurement sections 2001 is called "primary 
information", and the location and posture, and 
velocity and acceleration of the hand with reference to 

15 the head are called "secondary information". 

It is checked in step S8 if "secondary 
information" satisfies a predetermined state transition 
condition so as to determine transition of state (p. 
Then, the state transits to another state corresponding 

20 to the transition condition in which the "secondary 
information" is satisfied (step S10). After the 
transition, the current status, time elapsed from the 
previous state transition, and current location/posture 
of the hand and head are outputted (step S12) . 

25 The operation of the action analysis section 2002 

upon applying the functions shown in Figs. 7 and 8 to 



- 16 - 



an action (shooting) game will be explained below. In 
this shooting game, the following five states are 
defined, and values 0 to 4 are assigned in advance to 
these states. 

0 = initial state 

1 = loading (load a gun) state 

2 = shooting state 

3 = defense (protect a player) state 

4 = operation guidance state 

In this game apparatus, an action is expressed by a 
change in state value. As shown in Fig. 9, actions 
defined in this game apparatus are as follows: 

Transition Action 

* — > * No action (no change in state value) 



0 -> 1 



Loading action 



1 -> 0 



Loading cancel action 



1 



-» 2 



Firing action 



0^3 



Defense action 



3^0 



Defense cancel action 



0 -> 4 



Operation guidance action 



4 -> 0 



Operation guidance cancel action 



Note that the initial state is restored a predetermined 
period of time elapsed after the firing state. The 
output from the action analysis section 2002 includes 



action state cp and three-dimensional information of the 
head and hand. 

Fig. 10 is a flow chart showing the overall state 
transition condition determination process of the 
5 action analysis section 2002. More specifically, the 

action analysis section 2002 executes one of steps S100, 
S200, S300, S400, and S500 (respectively shown in 
detail in Figs. 11, 12, 13, 14, and 15) in accordance 
with the current state value (p. Each of steps S100, 

10 S200, S300, S400, and S500 outputs the action state 
value (p after transition, and three-dimensional 
information of the head and hand. 

The control sequence when the current state value 
= 0, i.e., "initial state" is currently set, will be 

15 explained below with reference to Fig. 11. There are 
four states to which the initial state can transit, 
i.e., state 0 (no transition), state 1 (loading state), 
state 3 (defense state) , and state 4 (operation 
guidance state) , in the game apparatus of this 

20 embodiment. In the control sequence shown in Fig. 11, 
to which of the four states the initial state is to 
transit is determined in consideration of the amount of 
backward movement of the hand with respect to the head 
in the back (z) direction, the deviation of the angle 

25 the back-and-f orth direction (z-direction in Fig. 6) of 
the hand makes with the back direction (z-direction in 
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Fig. 5) of the player's head, the deviation of the 
angle the back of the hand {y-direction in Fig. 6) 
makes with the back direction of the head, and the like. 
Hence, the following variables z hand , CL lr and a 2 must be 
5 considered in the control sequence shown in Fig. 11. 

z hand : z-coordinate value of hand location with 
reference to head coordinate system (see Fig. 5) 

a x : angle z-axis of head coordinate system makes 
with that of hand coordinate system (see Fig. 6) 
10 a 2 : angle z-axis of head coordinate system makes 

with y-axis of hand coordinate system 

Assume that constant values C 01 , C 02 , C 03 , and C 04 
are respectively: 

C 01 = 0.0 (length) 
15 C 02 = 45° 

C 03 = 150° 

C 04 = 30° 

Then, if YES in steps S102 and S104, i.e., 

Zhand > C 01 and a x < C 02 

20 in other words, if the player has moved the hand 

backward to a location behind the center of the head 
(C 01 =0.0) in the back direction of the head while the 
deviation of the angle the direction of the hand makes 
with the back direction of the head is kept suppressed 

25 to be equal to or smaller than 45° (Fig. 16) , it is 
determined that the player wants to make a loading 
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action. Then, the current time t is saved in a 
register t 01 in step S106, the current hand location 
L hand is saved in a register L 01 in step S108, and an 
action state value = 1 (loading state) is output in 
5 step S110. 

On the other hand, if NO in steps S102 and S104, 
and YES in step S112, i.e., 

Zhand ^ C 01 and a x > C 02 and a 2 > C 03 
in other words, if the player has moved the hand to a 

10 location in front of the central position of the head 
(Fig. 17) while the deviation of the angle the 
direction of the hand makes with the back direction of 
the head is 45° or more, and an angle the direction 
perpendicular to the back of the hand makes with the 

15 back direction of the head is 150° or more, it is 
determined that the player wants to make a defense 
action. Then, the current time t is saved in a 
register t 03 in step S114, and an action state value = 
3 (defense state) is output in step S116. 

20 On the other hand, if NO in steps S102, S104, and 

S112, and YES in step S118, i.e., 

Zhand ^ C 01 and a x > C 02 and a 2 < C 04 
in other words, if the player has moved the hand to a 
location in front of the central position of the head 

25 while the deviation of the angle the direction of the 
hand makes with the back direction of the head is 45° 
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or more, and an angle the direction perpendicular to 
the back of the hand makes with the back direction of 
the head is 30° or less, it is determined that the 
player wants an operation guidance (i.e., HELP). Then, 
5 the current time t is saved in a register t 04 in step 

S120, and an action state value = 4 (operation guidance 
state) is output in step S122. 

On the other hand, if NO in steps S102, S104, 
S112, and S118, i.e. , 

10 z hand < C 01 and > C 02 and C 04 < a 2 < C 03 

it is determined that the player's action is not large 
enough to issue a command, and the initial state is 
maintained in step S124. 

The control sequence when the current state is 

15 the loading state (= 1) will be explained below with 

reference to Fig. 12. There are three states to which 
the loading state can transit, i.e., state 0 (initial 
state), state 1 (no transition), and state 2 (firing 
state)/ as shown in Fig. 9. In the control sequence 

20 shown in Fig. 12, a hand moving distance d, hand angle 
a 3 , and hand moving velocity v are referred to in 
addition to the aforementioned angle oq. Note that 

d: distance from hand location L 0 to current hand 
location L hand when state transits from 0 to 1 

25 a 3 : angle a velocity vector V of hand makes with 

z-axis of head coordinate system 
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v: moving velocity of hand 

Assume that constant values C n , C 12 , C 13 , and C 14 
are respectively: 

C n = 60° (to be compared with a x ) 
5 C 12 = 300 mm (to be compared with d) 

C 13 = 120° (to be compared with a 3 ) 

C 14 = 300 mm/s (to be compared with v) 
Then, if YES in step S201, i.e., 

«i > C n 

10 in other words, the deviation of the angle the hand 

direction makes with the back direction of the head has 
exceeded 60°, it is determined that the player has made 
a loading action but does not want to fire. Then, the 
current time t is saved in a register t 10 in step S202, 
15 and an action state value = 0 is output in step S204. 

On the other hand, if NO in step S201 and YES in 
steps S206, S208, and S210, i.e., 

a L < c n , d > c 12 , a 3 > c 13 , v > c 14 

in other words, if a change in direction of the hand 
20 with respect to the head is not large, the hand moving 
distance is equal to or larger than C 12 (e.g., 300 mm), 
the angle the hand moving direction makes with the back 
direction of the head is equal to or larger than the 
angle C 13 (e.g., 120°), and the hand moving velocity is 
25 equal to or higher than C 14 (e.g., 300 mm/s) (Fig. 18), 
it is determined that the player wants to fire. Then, 
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the current time t is saved in a register t 12 in step 
S212, and an action state value = 2 (firing state) is 
output in step S214. 

On the other hand, if the above conditions are 
5 not satisfied, i.e., 

a L < C n and d < C 12 or a 3 < C 13 , v < C 14 
the loading state is maintained. 

When the current state is the firing state (=2), 
the control sequence shown in Fig. 13 is executed. In 

10 the example shown in Fig. 9, the firing state can 

transit to either the initial state or firing state (no 
transition) . According to the control sequence shown 
in Fig. 13, if time At elapsed after the loading state 
transits to the firing state is smaller than a constant 

15 value T 2 (e.g., 200 ms) (NO in step S302), the firing 
state is maintained in step S308. On the other hand, 
if time At is equal to or longer than T 2 , the current 
time is saved in step S304, and the state is returned 
to the initial state to output that state value in step 

20 S306. 

In the example in Fig. 9, the initial state is 
unconditionally restored the predetermined period of 
time (T 2 ) after the beginning of shooting. Such simple 
state transition to the initial state is designed for 
25 the sake of simplicity upon explaining the rules of the 
game of the game apparatus of this embodiment. If the 
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game has a complex flow (scenario) , more states should 
be defined in correspondence with that complexity. In 
such case, more states in addition to the initial state 
may be defined as those to which the firing state can 
5 transit. 

When the current state is the defense state (= 3) , 
the control sequence shown in Fig. 14 is executed. In 
the example shown in Fig. 9, states to which the 
defense state can transit are only the initial state 

10 and the defense state (no transition) . According to 

the control sequence shown in Fig. 14, it is checked in 
step S402 if an angle the posture of the hand makes 
with that of the head is smaller than a predetermined 
value C 31 (e.g., 150°). That is, if 

15 ct 2 < C 31 

in other words, if an angle the direction of the back 
of the hand makes with the back direction of the head 
is set to be less than 150°, it is determined that the 
defense state is canceled, and the current state is 

20 returned to the initial state and the current time is 
saved in step S404. Furthermore, the initial state (= 
0) is output in step S406. 

On the other hand, if NO in step S402, but if it 
is determined in step S407 that time At elapsed after 

25 the previous state transited to the defense state is 
equal to or longer than a predetermined time T 3 , the 
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flow advances to step S404 to return the current state 
to the initial state like in the case wherein a 
sufficiently long time has elapsed from the firing 
state . 

5 Furthermore, if NO in steps S402 and S407, i.e., 

a 2 > C 31 and AT < T 3 
the defense state is maintained. 

If the state transits to "operation guidance 
state" in step S122 in Fig. 11, i.e., 

10 z hand < C 01 and a 2 > C 02 and a 2 < C 04 

in other words, if the player moves the hand to a 
location in front of the central position of the head 
in the back direction of the head while the deviation 
of the angle the direction of the hand makes with the 

15 back direction of the head is 45° or more, and an angle 
the direction perpendicular to the back of the hand 
makes with the back direction of the head is 30° or 
less, this means that the player is watching the back 
or joint of the hand. In such case, since a state 

20 value = 4 is output to the game apparatus 3000, the 
apparatus 3000 displays a HELP window on the 
predetermined display device. Note that this display 
device is preferably a head-mounted display device 
(HMD) . If the HMD is used, the player need not move 

25 the head upon watching the HELP window, and the control 
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sequence (Fig. 15) for maintaining the "operation 
guidance state" need not be complicated. 

Note that the HELP window may be either two- or 
three-dimensional display, or may be displayed in 
5 either a VR or AR environment, as long as it matches 

the current environment. In this embodiment, since the 
head location/posture sensor is attached to the player, 
it is preferable to display the HELP window in an AR 
environment in terms of efficient use of the sensor 100. 

10 With the above arrangement, a command obtained by 

analyzing the user's action in consideration of user's 
will is supplied to the game apparatus 3000, and an 
image corresponding to the command is displayed within 
the virtual space displayed on the display unit 4000. 

15 More specifically, a "target" is randomly 

displayed within the virtual space presented to the 
user by the game apparatus 3000, and the user shoots it 
by his or her actions. Since the user's action is 
reflected as a counteraction against the "target", the 

20 game with very high reality can be implemented. 

Various modifications of the present invention 
can be made . 

Modification 1: 

In the above embodiment, commands and states 
25 shown in Fig. 9 are used. However, the present 

invention is not limited to such specific commands and 
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states. That is, the user interface of the present 
invention is not limited to a game apparatus 
environment. The present invention can be applied to 
any other environments as long as the user's or 
5 player's actions are required to be used in place of 
commands (or instructions) upon presenting CG data to 
the user or player. 

For example, the present invention may be applied 
to a sign language recognition user interface, a user 

10 interface for manipulating home electronic products, a 
user interface for manipulating industrial machines 
(special vehicles for construction work, factory lines) , 
a user interface for physically handicapped person 
assist apparatuses (assist bed, motored wheelchair) , 

15 and the like. 

Modification 2: 

When the present invention is applied to a game 
apparatus, the game to which the present invention is 
applied is not limited to the rules shown in Fig. 9. 
20 The present invention can be applied to more or less 
states (commands) . Also, state transition is not 
limited to Fig. 9. 

Modification 3: 

In the game system of the above embodiment, since 
25 the action states have one-to-one correspondence with 
commands or instructions, the command conversion 
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section 2003 can directly convert an action state into 
a command. However, the present invention is not 
limited to such specific system. That is, the command 
conversion unit 2003 is designed to improve 
5 compatibility to the game apparatus 3000 connected to 
the unit 2000. In other words, the game apparatus 3000 
can be an existing versatile game apparatus. The 
existing game apparatus has an existing command system. 
The command conversion section 2003 is designed to have 

10 a table for receiving outputs (state, velocity, 

acceleration) from the action analysis section 2002, 
and converting them into a command system for the game 
apparatus 3000. This table is rewritable, and the 
table contents are rewritten in correspondence with the 

15 game apparatus 3000 used. In this way, the unit 2000 
can be applied to a plurality of different types of 
game apparatuses having different command systems 
without changing the arrangement and operation of the 
unit 2000. 

20 Modification 4: 

In the above embodiment, sensors are attached to 
the head and hand. However, the present invention is 
not limited to this. That is, to achieve the objective 
for issuing a command by an action, the sensor may be 

25 attached to the fingertip, arm, leg, thigh, knee, or 

shin in place of the hand. For example, as an example 
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of a sensor to be attached to finger joints, a so- 
called "glove type hand joint angle measurement device" 
has been put into practice. 
Modification 5: 

In the above embodiment, the command conversion 
section 2003 receives an action identifier from the 
analysis section 2002, and outputs a command signal 
corresponding to an action. Upon receiving this 
command, the command signal system of the programs and 
apparatuses (game apparatus and the like) that use the 
command signal is defined in the game apparatus 3000. 
However, the present invention is not limited to such 
specific command signal. As the command signal, the 
following formats are available. That is, 

•to output a command signal as an electrical or 
optical signal; 

•to implement command conversion as a function 
program on a computer, and output data to be written in 
a predetermined storage area by the function program to 
the conversion unit 2003 as a command signal; 

•to implement command conversion as a function 
program on a computer, and launch a callback function 
which corresponds to a command in advance to use it as 
a command signal output; and 

•to implement command conversion as a function 
program on a computer, and generate an interrupt signal 



which corresponds to a command in advance to use it as 
a command signal output . 
Modification 6: 

In the above embodiment, magnetic sensors are 
5 used as the sensors. Alternatively, the 

location/posture may be measured using an ultrasonic 
wave, mechanically, or by means of an image process. 

To restate, according to the present invention, a 
user interface apparatus, user interface method, and 
10 game apparatus, to which the user (player) can easily 
sensuously become accustomed, and which can accurately 
recognize instructions (commands) that the user 
(player) intended can be provided. 

As many apparently widely different embodiments 
15 of the present invention can be made without departing 
from the spirit and scope thereof, it is to be 
understood that the invention is not limited to the 
specific embodiments thereof except as defined in the 
appended claims . 
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WHAT IS CLAIMED IS: 

1. A user interface apparatus comprising: 

a first sensor attached to a first portion of a 
body of a user; 

a second sensor attached to a second portion 
different from the first portion; 

means for generating action information of the 
second portion on the basis of a relative position of 
the second portion with respect to the first portion,, 
which is detected by said first and second sensors; and 

determination means for determining a user 
instruction corresponding to the generated action 
information. 

2. The apparatus according to claim 1, wherein the 
first portion is a head. 

3. The apparatus according to claim 1, wherein the 
second portion is a hand. 

4. The apparatus according to claim 1, wherein said 
first sensor detects a location/posture of the first 
portion . 

5. The apparatus according to claim 1, wherein said 
second sensor detects a location and posture of the 
second portion. 

6. The apparatus according to claim 1, wherein the 
action information includes information which pertains 
to a state change of the second portion with respect to 



a location of the first portion, and information which 
pertains to a location change velocity of the state 
change . 

7. The apparatus according to claim 1, wherein the 
5 action information includes information which pertains 
to a state change of the second portion with respect to 
a location or location/posture of the first portion, 
and information which pertains to a location change 
acceleration of the state change. 
10 8. The apparatus according to claim 1, wherein the 
action information includes information which pertains 
to a posture of the second portion with respect to a 
posture of the first portion. 

9. The apparatus according to claim 1, wherein the 
15 action information includes information which pertains 

to a moving direction of a location of the second 
portion with respect to a posture of the first portion. 

10. The apparatus according to claim 1, further 
comprising : 

20 means for storing a value of the relative 

position of the second portion with respect to the 
first portion, and a plurality of state values which 
are defined in advance as a result of transition of the 
value; and 
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means for storing a plurality of different user 
instruction values corresponding to the plurality of 
state values. 

11. The apparatus according to claim 1, wherein said 
5 determination means decomposes the determined user 

instruction into a plurality of instruction operands, 
and outputs the operands. 

12. The apparatus according to claim 1, wherein when 
said generation means determines that a relative 

10 relationship between a location/posture of a head 

detected by said first sensor, and a location/posture 
of a hand detected by said second sensor indicates an 
action of the user whose line of sight is pointing to a 
predetermined portion of the hand, 

15 said determination means outputs a user 

instruction for outputting an operation guidance. 

13. The apparatus according to claim 1, further 
comprising a third sensor for detecting a bent angle of 
a finger. 

20 14. A game apparatus comprising: 

a first sensor for detecting a location/posture 
of a head of a player; 

a second sensor for detecting a location/posture 
of a hand or arm; 
25 means for estimating an action of the player on 

the basis of a relative location/posture of the hand or 
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arm with respect to the location/posture of the head, 
which are detected by said first and second sensors; 
and 

means for outputting a player command 
5 corresponding to the estimated action. 

15. The apparatus according to claim 14, further 
comprising display means for displaying an image of a 
game scene in front of the head of the player for the 
player . 

10 16. The apparatus according to claim 14, wherein said 
display means is a head-mounted display. 
17. The apparatus according to claim 14, wherein 
three steps including preparation, execution, and 
return steps, are prepared for the player command, and 

15 the steps are respectively defined as: 

a preparation action step for the player command 
when the location of the hand of the player moves to a 
position behind a position in front of a face of the 
player; 

20 an execution step for the player command when the 

location of the hand of the player moves forward from a 
rear position after the preparation action; and 

a return action step for the player command when 
the location of the hand of the player returns to a 

25 position of the face of the player after the forward 
movement . 
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18. A user interface method for outputting a user 
instruction to a predetermined apparatus or program, 
comprising : 

the step of detecting a location of a first 
5 portion of a body of a user and a location of a second 
portion different from the first portion using first 
and second sensors attached to the user; and 

the step of determining a user instruction on the 
basis of a relative position of the second portion with 
10 respect to the first portion, which are detected by 
said first and second sensors, and outputting the 
determined user instruction to the apparatus or program. 

19. The method according to claim 18, wherein said 
first and second sensors respectively detect 

15 locations/postures of the first and second portions. 

20. The method according to claim 18, wherein the 
first portion is a head of the user, and the second 
portion is a hand of the user. 

21. The method according to claim 18, further 

20 comprising the step of detecting information which 

pertains to a state change of the second portion with 
respect to a location or a location/posture of the 
first portion, and information which pertains to a 
location change velocity of the state change. 

25 22. The method according to claim 18, further 

comprising the step of detecting information which 
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pertains to a state change of the second portion with 
respect to a location or a location/posture of the 
first portion, and information which pertains to a 
location change acceleration of the state change. 
5 23. The method according to claim 18, further 
comprising the step of detecting a posture of the 
second portion with respect to a posture of the first 
portion . 

24. The method according to claim 18, further 

10 comprising the step of detecting a moving direction of 
a location of the second portion with respect to a 
posture of the first portion. 

25. The method according to claim 18, further 
comprising the step of storing a value of the relative 

15 position of the second portion with respect to the 
first portion, and a plurality of user instructions 
which are defined in advance as a result of transition 
of the value. 

26. The method according to claim 18, wherein the 
20 output step includes the step of decomposing the 

determined user instruction into a plurality of 
instruction operands, and outputting the operands. 

27. The method according to claim 18, wherein when it 
is determined that a relative relationship between a 

25 location/posture of a head detected by said first 

sensor, and a location/posture of a hand detected by 
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said second sensor indicates an action of the user 
whose line of sight is pointing to a predetermined 
portion of the hand, 

the output step includes the step of outputting a 
5 user instruction for outputting an operation guidance. 

28. The method according to claim 19, further 
comprising the step of detecting a bent angle of a 
finger from a third sensor. 

29. A computer readable storage medium, which stores 
10 a program of a user interface method for outputting a 

user instruction to a predetermined apparatus or 

program, storing: 

a program step of detecting a location of a first 

portion of a body of a user and a location of a second 
15 portion different from the first portion using first 

and second sensors attached to the user; and 

a program step of determining a user instruction 

on the basis of a relative position of the second 

portion with respect to the first portion, which are 
20 detected by said first and second sensors, and 

outputting the determined user instruction to the 

apparatus or program. 

30. A game apparatus for displaying a CG image in 
front of a field of view of a player, comprising: 

25 a first sensor for detecting a location of a 

first portion of a body of the player; 
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a second sensor for detecting a location of a 
second portion of the player, which is different from 
the first portion; and 

game progress means for proceeding with a game by 
5 determining a command the player wants to input on the 
basis of a relative position of the second portion with 
respect to the first portion, which are detected by 
said first and second sensors, and executing the 
determined command. 
10 31. The apparatus according to claim 30, wherein said 
first and second sensors respectively detect 
locations/postures of the first and second portions. 

32. The apparatus according to claim 30, wherein the 
first portion is a head of the player, and the second 

15 portion is a hand of the player. 

33. The apparatus according to claim 30, further 
comprising means for detecting information which 
pertains to a state change of the second portion with 
respect to the location or a location/posture of the 

20 first portion, and information which pertains to a 
location change velocity of the state change, and 
wherein the command is generated in further 
consideration of the detected velocity. 

34. The apparatus according to claim 30, further 
25 comprising means for detecting information which 

pertains to a state change of the second portion with 
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respect to the location or a location/posture of the 
first portion, and information which pertains to a 
location change acceleration of the state change, and 
wherein the command is generated in further 
5 consideration of the detected acceleration. 

35. The apparatus according to claim 30, further 
comprising means for detecting a posture of the second 
portion with respect to a posture of the first portion, 
and 

10 wherein the command is generated in further 

consideration of the detected posture of the second 
portion . 

36. The apparatus according to claim 30, further 
comprising means for detecting a moving direction of 

15 the location of the second portion with respect to a 
posture of the first portion, and wherein the command 
is generated in further consideration of the detected 
moving direction of the location of the second portion. 

37. The apparatus according to claim 30, further 

20 comprising a memory for storing a value of the relative 
position of the second portion with respect to the 
first portion, and a plurality of player commands which 
are defined in advance as a result of transition of the 
value . 

25 38. The apparatus according to claim 30, wherein said 
game progress means decomposes the determined player 
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command into a plurality of operands, and executes the 
command in accordance with the plurality of decomposed 
operands . 

39. The apparatus according to claim 33, wherein when 
5 a relative relationship between a location/posture of a 

head detected by said first sensor, and a 
location/posture of a hand detected by said second 
sensor indicates that a line of sight of the player is 
pointing to a predetermined portion of the hand, 
10 said game progress means generates a player 

command for outputting an operation guidance. 

40. The apparatus according to claim 30, further 
comprising a third sensor for detecting a bent angle of 
a finger. 

15 41. The apparatus according to claim 1, wherein the 
action information further includes geometric 
information of the first and second portions in a 
reference coordinate system. 

42. The apparatus according to claim 14, wherein said 
20 estimation means extracts geometric information of the 

first and second portions in a reference coordinate 
system, and estimates a player action on the basis of 
the geometric information. 

43. The method according to claim 18, wherein the 
25 detection step includes the step of extracting 
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geometric information of the first and second portions 
in a reference coordinate system. 

44. The apparatus according to claim 30, wherein said 
game progress means further extracts geometric 
5 information of the first and second portions in a 

reference coordinate system, estimates an action of the 
player on the basis of the geometric information, and 
determines the command the player wants to input on the 
basis of the estimated action. 

10 45. An interface apparatus comprising: 

a first sensor for detecting a location or motion 
of a first portion of a body of a user; 

a second sensor for detecting a location or 
motion of a second portion of the user, which is 

15 different from the first portions- 
determination means for analyzing outputs from 
said first and second sensors and determining a command 
input by the user on the basis of patterns of the 
outputs from said first and second sensors; and 

2 0 output means for outputting the command 

determined by said determination means to an object to 
be controlled. 

46. The apparatus according to claim 45, wherein the 
first portion is a head, and the second portion is a 
25 hand. 
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47. The apparatus according to claim 45, wherein said 
first and second sensors respectively detect locations 
and postures of the first and second portions. 

48. The apparatus according to claim 47, wherein said 
5 first and second sensors respectively detect location 

change velocities, location change accelerations, and 
moving directions of the first and second portions. 

49. The apparatus according to claim 48, wherein said 
determination means analyzes an action of -the user 

10 using information of a change in location, the location 
change velocity, location change acceleration, and 
relative moving direction of the second portion with 
respect to the first portion, and specifies a command 
corresponding to the action. 

15 50. The apparatus according to claim 49, wherein said 
determination means has storage information which 
defines in advance a relationship between the action of 
the user and the corresponding command. 

51. The apparatus according to claim 50, wherein the 
20 command output from said determination means is 

supplied to the object to be controlled, and an image 
corresponding to the command is displayed. 

52. A user interface method comprising: 

the first detection step of detecting a location 
25 or motion of a first portion of a body of a user; 
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the second detection step of detecting a location 
or motion of a second portion of the user, which is 
different from the first portion; 

the determination step of analyzing outputs of 
5 the first and second detection steps and determining a 
command input by the user on the basis of patterns of 
the outputs of the first and second detection steps; 
and 

the output step of outputting the command 
10 determined in the determination step to an object to be 
controlled. 

53. The method according to claim 52, wherein the 
first portion is a head, and the second portion is a 
hand. 

15 54. The method according to claim 52, wherein the 

first and second detection steps include the steps of 
respectively detecting locations and postures of the 
first and second portions. 

55. The method according to claim 54, wherein the 
20 first and second detection steps include the steps of 

respectively detecting location change velocities, 
location change accelerations, and moving directions of 
the first and second portions. 

56. The method according to claim 55, wherein the 
25 determination step includes the step of analyzing an 

action of the user using information of a change in 
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location, the location change velocity, location change 
acceleration, and relative moving direction of the 
second portion with respect to the first portion, and 
specifying a command corresponding to the action. 
5 57. The method according to claim 56, wherein the 

determination step includes the step of determining the 
command on the basis of storage information which 
defines in advance a relationship between the action of 
the user and the corresponding command. 
10 58. The method according to claim 57, wherein the 
object to be controlled displays an image corresponding 
to the command. 



ABSTRACT OF THE DISCLOSURE 
There is disclosed a user interface apparatus to 
which the user (player) can easily sensuoulsy become 
accustomed, and which can accurately recognize 
5 instructions (commands) that the user (player) intended. 
The user interface apparatus has a location/posture 
sensor (100) attached to the head of the user, and a 
location/posture sensor (200) attached to his or her 
hand, analyzes hand action on the basis of the relative 
10 position of the location/posture of the hand with 

respect to those of the head, and estimates the current 
action state from the analysis result. The apparatus 
then determines a user instruction (player command) on 
the basis of the action state. 
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As a below named inventor, I hereby declare that: 

My residence, post office address and citizenship are as stated below next to my name; 

I believe I am the original, first and sole inventor (if only one name is listed below) or an 
original, first and joint inventor (if plural names are listed below) of the subject matter which is claimed 
and for which a patent is sought on the invention entitled 

USER INTERFACE APPARATUS. USER INTERFACE METHOD , 

GAMF, APPARATUS. AND PROGRAM STORAGE MEDIUM 



the specification of which [X]is attached hereto. [ ] was filed on 

as United States Application No. or PCT International Application No. _ 

and was amended on (if applicable). 

I hereby state that I have reviewed and understand the contents of the above-identified specification, 
including the claims, as amended by any amendment referred to above. 
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CFR §1.56. 

I hereby claim foreign priority benefits under 35 U.S.C. §119(a)-(d) or §365(b), of any foreign 
application(s)for patent or inventor's certificate, or §365(a) of any PCT international application which 
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any foreign application for patent or inventor's certificate, or PCT international application having a filing 
date before that of the application on which priority is claimed: 
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Country Application No. Filed fDav/Mo./Yr.) Priority Claimed 

JAPAN 11-165709 11/06/1999 Yes 

I hereby appoint the practitioners associated with the firm and customer number provided below 
to prosecute this application and to transact all business in the Patent and Trademark Office connected 
therewith, and direct that all correspondence be addressed to the address associated with that Customer 
Number: 

FITZPATRICK, CELLA, HARPER & SCINTO 
Customer Number: 05514 

I hereby declare that all statements made herein of my own knowledge are true and that all 
statements made on information and belief are believed to be true; and further that these statements were 
made with the knowledge that willful false statements and the like so made are punishable by fine or 
imprisonment, or both, under Section 1001 of Title 18 of the United States Code and that such willful 
false statements may jeopardize the validity of the application or any patent issued thereon. - 
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